[jira] [Comment Edited] (CASSANDRA-16625) Add a CircleCI job to run some tests repeatedly

["David Capwell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326753#comment-17326753 ] 

David Capwell edited comment on CASSANDRA-16625 at 4/21/21, 5:28 PM:
---------------------------------------------------------------------

Good idea, it would replace my following scripts (which just runs the test in a loop till it fails)

{code}
$ cat ci-test-loop
#!/usr/bin/env bash

#set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

bin="$(cd "$(dirname "$0")" > /dev/null; pwd)"

class_name="$1"
path="$(echo "$class_name" | tr '.' '/' ).java"

if [[ ! "$path" =~ distributed/upgrade ]]; then
  ant realclean && ant && ant generate-idea-files
fi

counter=1
echo ">>> Running attempt $counter"
while $bin/ci-test "$class_name" --reuse; do
  counter=$(( $counter + 1 ))
  echo ">>> Running attempt $counter"
done;
{code}

{code}
$ cat ci-test
#!/usr/bin/env bash

#set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

usage() {
  if [[ $# -gt 0 ]]; then
    echo "$*" 1>&2
  fi
  cat <<EOF
Usage: $(basename $0) [class name] (options)*
Options:
--cdc           - runs with cdc enabled
--compression   - run with compression enabled
--reuse         - skip calling ant realclean
EOF
  exit 1
}

if [[ $# -eq 0 ]]; then
  usage "Missing required arguments"
fi

class_name="$1"
shift
path="$(echo "$class_name" | tr '.' '/' ).java"
prefix="$( find test | grep "$path" | awk -F/ '{print $2}' )"

task="testclasslist"
fresh_build=true
while [[ "$#" -gt 0 ]]; do
  case "$1" in
    --cdc)
      task="testclasslist-cdc"
      shift
      ;;
    --compression)
      task="testclasslist-compression"
      shift
      ;;
    --reuse)
      fresh_build=false
      shift
      ;;
    *)
      usage "Unknown argument $1"
      ;;
  esac
done

if [[ "$fresh_build" == true ]]; then
  if [[ ! "$path" =~ distributed/upgrade ]]; then
    ant realclean && ant && ant generate-idea-files
  fi
fi

# cleanup logs so w/e is around are for this run
rm -rf build/test/logs || true

test_timeout=$(grep "name=\"test.$prefix.timeout\"" build.xml | awk -F'"' '{print $4}' || true)
if [ -z "$test_timeout" ]; then
  test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
fi
ant "$task" -Dtest.timeout="$test_timeout" -Dtest.classlistfile=<(echo "$path") -Dtest.classlistprefix=$prefix
{code}


Is there a way to trigger a Circle CI build with provided params?  If so then this doesn't seem bad.


was (Author: dcapwell):
Good idea, it would replace my following scripts (which just runs the test in a loop till it fails)

{code}
$ cat ci-test-loop
#!/usr/bin/env bash

#set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

bin="$(cd "$(dirname "$0")" > /dev/null; pwd)"

class_name="$1"
path="$(echo "$class_name" | tr '.' '/' ).java"
#prefix="$( find test | grep "$path" | awk -F/ '{print $2}' )"

if [[ ! "$path" =~ distributed/upgrade ]]; then
  ant realclean && ant && ant generate-idea-files
fi

#test_timeout=$(grep "name=\"test.$prefix.timeout\"" build.xml | awk -F'"' '{print $4}' || true)
#if [ -z "$test_timeout" ]; then
#  test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
#fi

counter=1
echo ">>> Running attempt $counter"
while $bin/ci-test "$class_name" --reuse; do
  counter=$(( $counter + 1 ))
  echo ">>> Running attempt $counter"
done;
{code}

{code}
$ cat ci-test
#!/usr/bin/env bash

#set -o xtrace
set -o errexit
set -o pipefail
set -o nounset

usage() {
  if [[ $# -gt 0 ]]; then
    echo "$*" 1>&2
  fi
  cat <<EOF
Usage: $(basename $0) [class name] (options)*
Options:
--cdc           - runs with cdc enabled
--compression   - run with compression enabled
--reuse         - skip calling ant realclean
EOF
  exit 1
}

if [[ $# -eq 0 ]]; then
  usage "Missing required arguments"
fi

class_name="$1"
shift
path="$(echo "$class_name" | tr '.' '/' ).java"
prefix="$( find test | grep "$path" | awk -F/ '{print $2}' )"

task="testclasslist"
fresh_build=true
while [[ "$#" -gt 0 ]]; do
  case "$1" in
    --cdc)
      task="testclasslist-cdc"
      shift
      ;;
    --compression)
      task="testclasslist-compression"
      shift
      ;;
    --reuse)
      fresh_build=false
      shift
      ;;
    *)
      usage "Unknown argument $1"
      ;;
  esac
done

if [[ "$fresh_build" == true ]]; then
  if [[ ! "$path" =~ distributed/upgrade ]]; then
    ant realclean && ant && ant generate-idea-files
  fi
fi

# cleanup logs so w/e is around are for this run
rm -rf build/test/logs || true

test_timeout=$(grep "name=\"test.$prefix.timeout\"" build.xml | awk -F'"' '{print $4}' || true)
if [ -z "$test_timeout" ]; then
  test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
fi
ant "$task" -Dtest.timeout="$test_timeout" -Dtest.classlistfile=<(echo "$path") -Dtest.classlistprefix=$prefix
{code}


Is there a way to trigger a Circle CI build with provided params?  If so then this doesn't seem bad.

> Add a CircleCI job to run some tests repeatedly
> -----------------------------------------------
>
>                 Key: CASSANDRA-16625
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16625
>             Project: Cassandra
>          Issue Type: Task
>          Components: CI
>            Reporter: Andres de la Peña
>            Priority: Normal
>
> I think it could be useful to have an optional CircleCI job to run some specific tests n times. That way, tickets could attach CircleCI runs showing that the changes don't make a certain ticket flaky or, conversely, that they fix a flaky test. Doing this systematically should mitigate the risk of introducing new flaky tests, and I guess it would be more convenient and easy to share than running the tests locally or on a private CI system.
> It would also be nice to have something similar in Jenkins, but I'm focusing this ticket on CircleCI because it's available also for non-committers, so assignees can run their tests before setting the tickets as ready for review.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org