You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Semb Wever (Jira)" <ji...@apache.org> on 2020/08/01 09:23:00 UTC
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Semb Wever updated CASSANDRA-16006:
-------------------------------------------
Description:
Currently dtests in Jenkins take ~10 hours.
Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes.
This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29] with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration).
In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added.
The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations.
Splits can fail for silly reasons (false-positive), like [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits.
was:
Currently dtests in Jenkins take ~10 hours.
Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes.
This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29] with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration).
In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added.
The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations.
Splits can fail for silly reasons (false-positive), like full /tmp disks, or connectivity issues between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits.
> Parallelise Jenkins dtests
> --------------------------
>
> Key: CASSANDRA-16006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16006
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 4.0-rc
>
>
> Currently dtests in Jenkins take ~10 hours.
> Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes.
> This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29] with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration).
> In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added.
> The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations.
> Splits can fail for silly reasons (false-positive), like [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org