[jira] [Created] (FLINK-8745) Reduce travis usage

["Chesnay Schepler (JIRA)" <ji...@apache.org>]

Chesnay Schepler created FLINK-8745:
---------------------------------------

             Summary: Reduce travis usage
                 Key: FLINK-8745
                 URL: https://issues.apache.org/jira/browse/FLINK-8745
             Project: Flink
          Issue Type: Improvement
          Components: Build System, Travis
    Affects Versions: 1.4.0, 1.5.0
            Reporter: Chesnay Schepler
            Assignee: Chesnay Schepler


We've been notified by INFRA that our travis usage is exceedingly high.

There are various things we could look into short- and long term:

h2. Short-term

h3. Reduce number of jobs

We currently run 12 job for each pr/push.
The first 10 jobs belong to 2 groups, with each group representing one test run of Flink against a specific hadoop version.

Given that the majority of changes made to Flink do not impact our compatibility with hadoop we could drop one of the groups and instead rely on daily cron jobs. This alone would cut our travis usage by 40%.

Once the migration to flip6 is done we can drop the remaining 2 jobs, increasing the reduction to 60%.

h3. Reduce number of builds

Travis is run for every PR, regardless of what change was made, even if it was something trivial as removing a trailing space in a documentation file. From time to time it also happens that new commits are pushed in a PR solely to trigger a new build to get that perfect green build.

Instead we could look into manually triggering travis for pull requests, that is with a bot.

h2. Long-term

h3. Incremental builds

Making the build dependent on the changes made has been brought up a few times now. This would in particular benefit cases where connectors/libraries are modified as they generally have few dependents. We would still have to run everything though if changes are made to the core modules.

h3. Repository split

The most painful of them all, but in my opinion also the most promising. With separate repositories for the core flink modules (flink-runtime etc), flink-connectors and flink-libraries would cut downright skip the compilation for a large number of modules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)