[jira] [Created] (ARROW-16589) [CI][Dev] Make tasks.yml easier to maintain

["Krisztian Szucs (Jira)" <ji...@apache.org>]

Krisztian Szucs created ARROW-16589:
---------------------------------------

             Summary: [CI][Dev] Make tasks.yml easier to maintain
                 Key: ARROW-16589
                 URL: https://issues.apache.org/jira/browse/ARROW-16589
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Continuous Integration, Developer Tools
            Reporter: Krisztian Szucs


I think {{dev/tasks/tasks.yml}} has reached its limits by using jinja2 templated yml. 

We should think about a better way to define crossbow jobs while:
- keeping it readable
- in a dialect which is natively supported by editors
- while supporting tasks parametrization

Just one idea is to use python files containing python objects, e.g.:

{code}
Task(
  name="wheel-macos-big-sur-cp38-arm64",
  ci="github",
  template="python-wheels/github.osx.arm64.yml",
  params=dict(
    arch="arm64",
    arrow_simd_level="DEFAULT",
    python_version="3.8",
    macos_deployment_target="11.0"
  ),
  artifacts=[
    "pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
  ]
)
{code}

where {{Task}} would be the crossbow task class (which could be refactored to use pydantic or another alternative for less boilerplate). Of course porting to the tasks definitions to plain python could make the situation even worse by accessing too many scripting utilities. We could try a dynamic config language which sits between yaml and python like HCL.

[~kou] what syntax would you be comfortable to work with? Do you have any alternatives we could use?

cc [~amol-] [~raulcd] [~assignUser]




--
This message was sent by Atlassian Jira
(v8.20.7#820007)