You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ash Berlin-Taylor (JIRA)" <ji...@apache.org> on 2019/04/29 22:02:00 UTC
[jira] [Comment Edited] (AIRFLOW-2488) Speed up the CI
[ https://issues.apache.org/jira/browse/AIRFLOW-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829757#comment-16829757 ]
Ash Berlin-Taylor edited comment on AIRFLOW-2488 at 4/29/19 10:01 PM:
----------------------------------------------------------------------
I've just landed AIRFLOW-4358 and AIRFLOW-4394 which takes the total time now down to ~28min, and slowest individual tests are now:
[success] 4.34% tests.test_impersonation.ImpersonationTest.test_impersonation_subdag: {color:#FF0000}44.6452s{color}
[success] 2.93% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0: {color:#FF0000}30.0785s{color}
[success] 2.36% tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock: {color:red}24.2686s{color}
[success] 2.10% tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies: {color:red}21.5965s{color}
[success] 1.96% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards: {color:red}20.1814s{color}
[success] 1.96% tests.test_impersonation.ImpersonationTest.test_impersonation_custom: {color:red}20.1730s{color}
[success] 1.96% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672: {color:red}20.1652s{color}
[success] 1.78% tests.cli.test_cli.TestCLI.test_next_execution: {color:red}18.3203s{color}
[success] 1.47% tests.sensors.test_timeout_sensor.SensorTimeoutTest.test_timeout: {color:red}15.1303s{color}
[success] 1.27% tests.transplant_class.<locals>.C.test_cli_webserver_shutdown_when_gunicorn_master_is_killed: {color:red}13.0136s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_no_impersonation: {color:red}11.6442s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_impersonation: {color:red}11.5675s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_default_impersonation: {color:red}11.5617s{color}
The "unit" tests now run in 25mins and the kube tests take 28mins, so speeding up those would be worth it
was (Author: ashb):
I've just landed AIRFLOW-4358 and AIRFLOW-4393 which takes the total time now down to ~28min, and slowest individual tests are now:
[success] 4.34% tests.test_impersonation.ImpersonationTest.test_impersonation_subdag: {color:#FF0000}44.6452s{color}
[success] 2.93% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0: {color:#FF0000}30.0785s{color}
[success] 2.36% tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock: {color:red}24.2686s{color}
[success] 2.10% tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies: {color:red}21.5965s{color}
[success] 1.96% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards: {color:red}20.1814s{color}
[success] 1.96% tests.test_impersonation.ImpersonationTest.test_impersonation_custom: {color:red}20.1730s{color}
[success] 1.96% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672: {color:red}20.1652s{color}
[success] 1.78% tests.cli.test_cli.TestCLI.test_next_execution: {color:red}18.3203s{color}
[success] 1.47% tests.sensors.test_timeout_sensor.SensorTimeoutTest.test_timeout: {color:red}15.1303s{color}
[success] 1.27% tests.transplant_class.<locals>.C.test_cli_webserver_shutdown_when_gunicorn_master_is_killed: {color:red}13.0136s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_no_impersonation: {color:red}11.6442s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_impersonation: {color:red}11.5675s{color}
[success] 1.13% tests.test_impersonation.ImpersonationTest.test_default_impersonation: {color:red}11.5617s{color}
The "unit" tests now run in 25mins and the kube tests take 28mins, so speeding up those would be worth it
> Speed up the CI
> ---------------
>
> Key: AIRFLOW-2488
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2488
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Fokko Driesprong
> Priority: Major
>
> Some of the tests take far to long:
> {noformat}
> [success] 10.48% tests.test_jobs.BackfillJobTest.test_backfill_examples: 177.9813s
> [success] 6.24% tests.test_jobs.BackfillJobTest.test_backfill_multi_dates: 106.0157s
> [success] 3.59% tests.transplant_class.<locals>.C.test_backfill: 61.0503s
> [success] 3.55% tests.contrib.sensors.test_python_sensor.PythonSensorTests.test_python_sensor_false: 60.2099s
> [success] 2.66% tests.test_impersonation.ImpersonationTest.test_impersonation_subdag: 45.1117s
> [success] 1.86% tests.test_jobs.SchedulerJobTest.test_dagrun_root_after_dagrun_unfinished: 31.5533s
> [success] 1.77% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_1_redis_redis_6379_0: 30.0873s
> [success] 1.48% tests.operators.test_subdag_operator.SubDagOperatorTests.test_subdag_deadlock: 25.0747s
> [success] 1.40% tests.test_jobs.SchedulerJobTest.test_new_import_error_replaces_old: 23.7093s
> [success] 1.36% tests.test_jobs.SchedulerJobTest.test_dagrun_success: 23.1645s
> [success] 1.33% tests.test_jobs.BackfillJobTest.test_cli_backfill_depends_on_past_backwards: 22.5161s
> [success] 1.30% tests.test_jobs.SchedulerJobTest.test_dagrun_deadlock_ignore_depends_on_past: 22.1402s
> [success] 1.30% tests.test_jobs.SchedulerJobTest.test_dagrun_deadlock_ignore_depends_on_past_advance_ex_date: 22.1153s
> [success] 1.28% tests.test_jobs.BackfillJobTest.test_cli_backfill_depends_on_past: 21.7944s
> [success] 1.27% tests.test_jobs.SchedulerJobTest.test_dagrun_root_fail_unfinished: 21.5010s
> [success] 1.25% tests.test_jobs.BackfillJobTest.test_run_ignores_all_dependencies: 21.3037s
> [success] 1.25% tests.test_jobs.SchedulerJobTest.test_dagrun_root_fail: 21.1455s
> [success] 1.20% tests.test_impersonation.ImpersonationTest.test_impersonation_custom: 20.3233s
> [success] 1.19% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration_0_amqp_guest_guest_rabbitmq_5672: 20.1648s
> [success] 1.14% tests.test_jobs.BackfillJobTest.test_backfill_run_backwards: 19.4043s
> [success] 1.06% tests.cli.test_cli.TestCLI.test_next_execution: 17.9290s
> {noformat}
> (List updated April 23, 2019)
> These tests should be refactored to make them faster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)