[jira] [Created] (FLINK-11208) Decrease build/testing time of modules

["Till Rohrmann (JIRA)" <ji...@apache.org>]

Till Rohrmann created FLINK-11208:
-------------------------------------

             Summary: Decrease build/testing time of modules
                 Key: FLINK-11208
                 URL: https://issues.apache.org/jira/browse/FLINK-11208
             Project: Flink
          Issue Type: Improvement
    Affects Versions: 1.8.0
            Reporter: Till Rohrmann


This issue is intended to be the umbrella issue for speeding up build and test times of our modules. At the moment building Flink takes more than 1 hour on my machine. Here are the modules sorted in descending order with respect to build+test times (second column in seconds):

{code}
flink-connector-kafka-0.11 766
flink-tests 425
flink-runtime 391
flink-yarn-tests 331
flink-table 207
flink-connector-kafka-0.9 189
flink-connector-cassandra 135
flink-connector-kafka-0.8 132
flink-connector-kafka-0.10 130
flink-gelly-examples 76
flink-hbase 60
flink-core 58.251
flink-storm-examples 56.796
flink-streaming-java 56.080
flink-ml 53.349
flink-connector-filesystem 47.698
flink-connector-elasticsearch5 47.143
flink-connector-elasticsearch2 45.406
flink-connector-elasticsearch 40.804
flink-streaming-scala 40.493
flink-storm 40.107
flink-scala-shell 38.566
flink-scala 34.964
flink-gelly 33.714
flink-fs-tests 30.466
flink-python 26.919d 
flink-java 24.899
flink-gelly-scala 23.406
flink-dist 21.778
flink-mesos 21.524
flink-hadoop-compatibility 20.106
flink-examples-streaming 19.951
flink-shaded-hadoop2 19.782
flink-s3-fs-presto 18.731
flink-s3-fs-hadoop 17.070
flink-optimizer 17.034
flink-hadoop-fs 15.857
flink-avro 15.409
flink-shaded-yarn-tests 15.147
flink-yarn 14.962
flink-swift-fs-hadoop 13.709
flink-streaming-python 13.163
flink-connector-wikiedits 12.515
flink-queryable-state-runtime 12.474
flink-statebackend-rocksdb 11.987
flink-shaded-hadoop2-uber 11.527
flink-connector-elasticsearch6 10.880
flink-runtime-web 10.384
flink-orc 9.916
flink-cep 8.963
flink-sql-client 8.944
flink-examples-batch 8.555
flink-cep-scala 8.270
flink-clients 5.966
flink-examples-table 5.769
flink-connector-kafka-base 5.754
flink-high-parallelism-iterations-test 4.707
flink-parquet 3.640
flink-hcatalog 3.436
flink-jdbc 3.296
flink 3.257
flink-connector-rabbitmq 2.779
flink-test-utils 2.686
flink-metrics-statsd 2.667
flink-elasticsearch6-test 2.251
flink-metrics-jmx 2.167
flink-elasticsearch5-test 2.144
flink-elasticsearch2-test 2.114
flink-metrics-dropwizard 1.898
flink-connector-elasticsearch-base 1.726
flink-metrics-datadog 1.676
flink-elasticsearch1-test 1.652
flink-mapr-fs 1.551
flink-json 1.526
flink-docs 1.441
flink-metrics-core 1.373
flink-avro-confluent-registry 1.345
flink-metrics-prometheus 1.340
flink-connector-twitter 1.276
flink-annotations 1.183
flink-queryable-state-test 1.121
force-shading 1.114
flink-queryable-state-client-java 1.085
flink-test-utils-junit 1.058
flink-sql-client-test 1.049
flink-confluent-schema-registry 1.001
flink-quickstart 0.939
flink-datastream-allround-test 0.924
flink-container 0.740
flink-stream-state-ttl-test 0.679
flink-metrics-slf4j 0.663
flink-stream-stateful-job-upgrade-test 0.623
flink-shaded-curator 0.544
flink-connector-nifi 0.487
flink-quickstart-java 0.451
flink-bucketing-sink-test 0.369
flink-local-recovery-and-allocation-test 0.220
flink-stream-sql-test 0.191
flink-quickstart-test 0.191
flink-streaming-file-sink-test 0.182
flink-dataset-allround-test 0.175
flink-distributed-cache-via-blob 0.163
flink-metrics-graphite 0.160
flink-examples 0.155
flink-parent-child-classloading-test 0.154
flink-quickstart-scala 0.135
flink-shaded-hadoop 0.120
flink-filesystems 0.081
flink-metrics 0.076
flink-test-utils-parent 0.074
flink-queryable-state 0.067
flink-formats 0.061
flink-state-backends 0.058
flink-end-to-end-tests 0.058
flink-libraries 0.056
flink-connectors 0.054
flink-contrib 0.052
{code}

As one can see {{flink-connector-kafka-0.11}} alone takes more than 12 minutes to build and run all tests. I think it would be good to revisit some of these tests to see whether they can be sped up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)