You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/05/29 13:30:02 UTC
Review Request 34804: Work with json module takes too long
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34804/
-----------------------------------------------------------
Review request for Ambari and Dmitro Lisnichenko.
Bugs: AMBARI-11529
https://issues.apache.org/jira/browse/AMBARI-11529
Repository: ambari
Description
-------
Due to profiling below, from 420 seconds of deploy json.load and json.loads
take 40 seconds (which even doesn't include working with json in ambari-agent
itself, only script):
204 0.007 0.000 20.182 0.099 __init__.py:243(load)
351 0.002 0.000 17.894 0.051 __init__.py:270(loads)
This is because json module for Python 2.6 is extremety slow. There are
benchmark for it on here:
<http://stackoverflow.com/questions/706101/python-json-decoding-performance>
due to it simplejson is at least ~40 times faster for deserializing.
Also another thing to do in scope of this jira is to replace inspect module
usage with os._getframe() which doesn't use realpath function that
intensively, taking away 10 more seconds.
30049/21438 1.712 0.000 10.873 0.001 posixpath.py:354(realpath)
Diffs
-----
LICENSE.txt e226388
NOTICE.txt 9e59af8
ambari-agent/conf/unix/install-helper.sh 7c18b35
ambari-agent/pom.xml 9a58537
ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34
ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362
ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae
ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20
ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230
ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568
ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07
ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196
ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323
ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3
ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85
ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027
ambari-agent/src/main/python/ambari_agent/hostname.py 9cb75d6
ambari-agent/src/main/python/ambari_agent/security.py dbb2192
ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8
ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c
ambari-agent/src/test/python/ambari_agent/TestController.py e00e737
ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb
ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1
ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION
ambari-common/src/main/python/resource_management/core/logger.py c741327
ambari-common/src/main/python/resource_management/core/shell.py 71b6501
ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py a66a25a
ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py 94f96ca
ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py eab01ad
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py b6be4c2
ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py 1e92f9d
ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8
ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87
ambari-server/conf/unix/install-helper.sh 108f4c2
ambari-server/pom.xml 4158d95
ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea
ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd
ambari-server/src/main/python/upgradeHelper.py 82c76bc
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py b4cff05
ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py 5ba5b03
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ee95493
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py 7b20a79
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py 40b5694
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 864961e
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 229607d
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 38270e8
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py df93633
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 6e12dd0
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py b5cd909
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 9028800
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 2846919
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py 3f87694
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py 80749e6
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py 45c8ea6
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py 073371a
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py 8e378b5
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b
ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py 35267db
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 32efb5c
Diff: https://reviews.apache.org/r/34804/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 34804: Work with json module takes too long
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34804/
-----------------------------------------------------------
(Updated May 29, 2015, 11:53 a.m.)
Review request for Ambari and Dmitro Lisnichenko.
Bugs: AMBARI-11529
https://issues.apache.org/jira/browse/AMBARI-11529
Repository: ambari
Description
-------
Due to profiling below, from 420 seconds of deploy json.load and json.loads
take 40 seconds (which even doesn't include working with json in ambari-agent
itself, only script):
204 0.007 0.000 20.182 0.099 __init__.py:243(load)
351 0.002 0.000 17.894 0.051 __init__.py:270(loads)
This is because json module for Python 2.6 is extremety slow. There are
benchmark for it on here:
<http://stackoverflow.com/questions/706101/python-json-decoding-performance>
due to it simplejson is at least ~40 times faster for deserializing.
Also another thing to do in scope of this jira is to replace inspect module
usage with os._getframe() which doesn't use realpath function that
intensively, taking away 10 more seconds.
30049/21438 1.712 0.000 10.873 0.001 posixpath.py:354(realpath)
Diffs (updated)
-----
LICENSE.txt e226388
NOTICE.txt 9e59af8
ambari-agent/conf/unix/install-helper.sh 7c18b35
ambari-agent/pom.xml 9a58537
ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34
ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362
ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae
ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20
ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230
ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568
ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07
ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196
ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323
ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3
ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85
ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027
ambari-agent/src/main/python/ambari_agent/security.py dbb2192
ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8
ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c
ambari-agent/src/test/python/ambari_agent/TestController.py e00e737
ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb
ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1
ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION
ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION
ambari-common/src/main/python/resource_management/core/logger.py c741327
ambari-common/src/main/python/resource_management/core/shell.py 71b6501
ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py a66a25a
ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py 94f96ca
ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py eab01ad
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py b6be4c2
ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py 1e92f9d
ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8
ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87
ambari-server/conf/unix/install-helper.sh 108f4c2
ambari-server/pom.xml 4158d95
ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea
ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd
ambari-server/src/main/python/upgradeHelper.py 82c76bc
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py b4cff05
ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py 5ba5b03
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ee95493
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py 7b20a79
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py 40b5694
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 864961e
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 229607d
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 38270e8
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py df93633
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 6e12dd0
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py b5cd909
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 9028800
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 2846919
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py 3f87694
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py 80749e6
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py 45c8ea6
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py 073371a
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py 8e378b5
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b
ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py 35267db
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 32efb5c
Diff: https://reviews.apache.org/r/34804/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 34804: Work with json module takes too long
Posted by Jeff Sposetti <je...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34804/#review85707
-----------------------------------------------------------
NOTICE.txt
<https://reviews.apache.org/r/34804/#comment137446>
Since this is dual licensed, we can elect to use MIT. So you can say this instead...
This product includes Simplejson, library fast encoding and decoding of json. (https://github.com/simplejson/simplejson - MIT license)
- Jeff Sposetti
On May 29, 2015, 11:29 a.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34804/
> -----------------------------------------------------------
>
> (Updated May 29, 2015, 11:29 a.m.)
>
>
> Review request for Ambari and Dmitro Lisnichenko.
>
>
> Bugs: AMBARI-11529
> https://issues.apache.org/jira/browse/AMBARI-11529
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Due to profiling below, from 420 seconds of deploy json.load and json.loads
> take 40 seconds (which even doesn't include working with json in ambari-agent
> itself, only script):
>
>
>
> 204 0.007 0.000 20.182 0.099 __init__.py:243(load)
> 351 0.002 0.000 17.894 0.051 __init__.py:270(loads)
>
>
> This is because json module for Python 2.6 is extremety slow. There are
> benchmark for it on here:
> <http://stackoverflow.com/questions/706101/python-json-decoding-performance>
>
> due to it simplejson is at least ~40 times faster for deserializing.
>
> Also another thing to do in scope of this jira is to replace inspect module
> usage with os._getframe() which doesn't use realpath function that
> intensively, taking away 10 more seconds.
>
>
>
> 30049/21438 1.712 0.000 10.873 0.001 posixpath.py:354(realpath)
>
>
> Diffs
> -----
>
> LICENSE.txt e226388
> NOTICE.txt 9e59af8
> ambari-agent/conf/unix/install-helper.sh 7c18b35
> ambari-agent/pom.xml 9a58537
> ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34
> ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362
> ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae
> ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20
> ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230
> ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568
> ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196
> ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323
> ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3
> ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85
> ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027
> ambari-agent/src/main/python/ambari_agent/hostname.py 9cb75d6
> ambari-agent/src/main/python/ambari_agent/security.py dbb2192
> ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8
> ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c
> ambari-agent/src/test/python/ambari_agent/TestController.py e00e737
> ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb
> ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1
> ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION
> ambari-common/src/main/python/resource_management/core/logger.py c741327
> ambari-common/src/main/python/resource_management/core/shell.py 71b6501
> ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py a66a25a
> ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py 94f96ca
> ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa
> ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py eab01ad
> ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py b6be4c2
> ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py 1e92f9d
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8
> ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87
> ambari-server/conf/unix/install-helper.sh 108f4c2
> ambari-server/pom.xml 4158d95
> ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea
> ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd
> ambari-server/src/main/python/upgradeHelper.py 82c76bc
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py b4cff05
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py 5ba5b03
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ee95493
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py 7b20a79
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py 40b5694
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 864961e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 229607d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 38270e8
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py df93633
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 6e12dd0
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py b5cd909
> ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 9028800
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 2846919
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py 3f87694
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py 80749e6
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py 45c8ea6
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py 073371a
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py 8e378b5
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b
> ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py 35267db
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 32efb5c
>
> Diff: https://reviews.apache.org/r/34804/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>
Re: Review Request 34804: Work with json module takes too long
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34804/#review85709
-----------------------------------------------------------
Ship it!
ambari-agent/src/main/python/ambari_agent/hostname.py
<https://reviews.apache.org/r/34804/#comment137448>
might not work when reverse DNS zone is not set up (e.g. on some OpenStack deployments). Let's just cache lookup result
- Dmitro Lisnichenko
On May 29, 2015, 11:29 a.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34804/
> -----------------------------------------------------------
>
> (Updated May 29, 2015, 11:29 a.m.)
>
>
> Review request for Ambari and Dmitro Lisnichenko.
>
>
> Bugs: AMBARI-11529
> https://issues.apache.org/jira/browse/AMBARI-11529
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Due to profiling below, from 420 seconds of deploy json.load and json.loads
> take 40 seconds (which even doesn't include working with json in ambari-agent
> itself, only script):
>
>
>
> 204 0.007 0.000 20.182 0.099 __init__.py:243(load)
> 351 0.002 0.000 17.894 0.051 __init__.py:270(loads)
>
>
> This is because json module for Python 2.6 is extremety slow. There are
> benchmark for it on here:
> <http://stackoverflow.com/questions/706101/python-json-decoding-performance>
>
> due to it simplejson is at least ~40 times faster for deserializing.
>
> Also another thing to do in scope of this jira is to replace inspect module
> usage with os._getframe() which doesn't use realpath function that
> intensively, taking away 10 more seconds.
>
>
>
> 30049/21438 1.712 0.000 10.873 0.001 posixpath.py:354(realpath)
>
>
> Diffs
> -----
>
> LICENSE.txt e226388
> NOTICE.txt 9e59af8
> ambari-agent/conf/unix/install-helper.sh 7c18b35
> ambari-agent/pom.xml 9a58537
> ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34
> ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362
> ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae
> ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20
> ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230
> ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568
> ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196
> ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323
> ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3
> ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85
> ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027
> ambari-agent/src/main/python/ambari_agent/hostname.py 9cb75d6
> ambari-agent/src/main/python/ambari_agent/security.py dbb2192
> ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8
> ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c
> ambari-agent/src/test/python/ambari_agent/TestController.py e00e737
> ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb
> ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1
> ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION
> ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION
> ambari-common/src/main/python/resource_management/core/logger.py c741327
> ambari-common/src/main/python/resource_management/core/shell.py 71b6501
> ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py a66a25a
> ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py 94f96ca
> ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa
> ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py eab01ad
> ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py b6be4c2
> ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py 1e92f9d
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8
> ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87
> ambari-server/conf/unix/install-helper.sh 108f4c2
> ambari-server/pom.xml 4158d95
> ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea
> ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd
> ambari-server/src/main/python/upgradeHelper.py 82c76bc
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py b4cff05
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py 5ba5b03
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ee95493
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py 7b20a79
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py 40b5694
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 864961e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 229607d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 38270e8
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py df93633
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 6e12dd0
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py b5cd909
> ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 9028800
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 2846919
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py 3f87694
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py 80749e6
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py 45c8ea6
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py 073371a
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py 8e378b5
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b
> ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py 35267db
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 32efb5c
>
> Diff: https://reviews.apache.org/r/34804/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>