You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Kouhei Sutou (Jira)" <ji...@apache.org> on 2022/05/17 00:03:00 UTC

[jira] [Commented] (ARROW-16589) [CI][Dev] Make tasks.yml easier to maintain

    [ https://issues.apache.org/jira/browse/ARROW-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537845#comment-17537845 ] 

Kouhei Sutou commented on ARROW-16589:
--------------------------------------

I'm OK with Python. I prefer external DSL such as HCL to internal DSL such as Python because external DSL sometimes lacks dynamic functionalities.

I want to add the following to requirements:

* Easy to write syntax (e.g.: less code, less magic and so on)

> [CI][Dev] Make tasks.yml easier to maintain
> -------------------------------------------
>
>                 Key: ARROW-16589
>                 URL: https://issues.apache.org/jira/browse/ARROW-16589
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Continuous Integration, Developer Tools
>            Reporter: Krisztian Szucs
>            Priority: Major
>
> I think {{dev/tasks/tasks.yml}} has reached its limits by using jinja2 templated yml. 
> We should think about a better way to define crossbow jobs while:
> - keeping it readable
> - in a dialect which is natively supported by editors
> - while supporting tasks parametrization
> Just one idea is to use python files containing python objects, e.g.:
> {code}
> Task(
>   name="wheel-macos-big-sur-cp38-arm64",
>   ci="github",
>   template="python-wheels/github.osx.arm64.yml",
>   params=dict(
>     arch="arm64",
>     arrow_simd_level="DEFAULT",
>     python_version="3.8",
>     macos_deployment_target="11.0"
>   ),
>   artifacts=[
>     "pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
>   ]
> )
> {code}
> where {{Task}} would be the crossbow task class (which could be refactored to use pydantic or another alternative for less boilerplate). Of course porting to the tasks definitions to plain python could make the situation even worse by accessing too many scripting utilities. We could try a dynamic config language which sits between yaml and python like HCL.
> [~kou] what syntax would you be comfortable to work with? Do you have any alternatives we could use?
> cc [~amol-] [~raulcd] [~assignUser]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)