You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "vbhanuchander-lang (via GitHub)" <gi...@apache.org> on 2026/08/09 23:18:33 UTC
[PR] feat(github): collect GitHub issue fields and map them onto domain issues [devlake]
vbhanuchander-lang opened a new pull request, #9038:
URL: https://github.com/apache/devlake/pull/9038
Closes #8840
### Summary
[GitHub issue fields](https://github.blog/changelog/2026-07-02-issue-fields-are-now-generally-available/) are organization-level structured issue metadata — typed, mutually exclusive within a field, and shared across every repository in the org. They went **generally available on 2026-07-02** (the issue was filed while they were still in preview).
That is precisely the gap @oliviertassinari described in #8840: `type:` labels are not mutually exclusive (`type: bug` and `type: regression` can both be applied) and are repo-scoped, so cross-org reporting is unreliable.
This PR collects issue field values and lets a scope config map a field onto an issue column, where it takes precedence over the existing label regexes.
### What's included
**New table `_tool_github_issue_field_values`** — one row per issue per field. `value` holds a queryable text form regardless of the field's type, and `raw_value` keeps the original JSON so anything that doesn't fit the text form (a `multi_select` array) is still recoverable:
| data_type | `value` |
|---|---|
| `single_select` | the selected option name |
| `multi_select` | option names, comma separated, in API order |
| `text` | the text |
| `number` | formatted without a trailing `.0` when integral, so an effort of 5 reads as `5` |
| `date` | as returned (`YYYY-MM-DD`) |
**New subtasks** `Collect Issue Field Values` and `Extract Issue Field Values`, hitting `GET /repos/{owner}/{repo}/issues/{issue_number}/issue-field-values`. Both are `EnabledByDefault: false`, so existing pipelines are untouched until someone opts in.
**New scope config keys**, each holding a field *name*:
`issueFieldPriority`, `issueFieldSeverity`, `issueFieldComponent`, `issueFieldStoryPoint`, `issueFieldDueDate`
When a mapping is set and the issue has a value for that field, it wins over the label regex. Field names are matched case-insensitively. `Effort` → `story_point` and `Target date` → `due_date` line up with the four fields GitHub preconfigures out of the box (Priority, Effort, Start date, Target date).
### Two design decisions worth your attention
**1. The mapping is applied in the convertor, not written back to `_tool_github_issues`.**
My first version added an `Enrich Issue Fields` subtask writing into `_tool_github_issues`, mirroring how the label regexes work today in `issue_extractor.go`. The table sorter rejected it, correctly:
```
cyclic dependency detected: map[Collect Issue Field Values:[Enrich Issue Fields] ...
Enrich Issue Fields:[Extract Issue Field Values Enrich Issue Fields] ...]
```
The collector iterates `_tool_github_issues` to build its request URLs, so anything that both reads and writes that table closes a loop. Applying the mapping in `ConvertIssues` removes the cycle, keeps the tool layer as raw GitHub truth, and avoids adding columns to `_tool_github_issues` at all. `fieldMapping.asSubtaskConfig()` is passed as `SubtaskConfig` so changing or clearing a mapping forces a full re-run rather than being skipped incrementally.
**2. A 404 is treated as "no field values", not an error.**
An org that has never configured issue fields, or a token without visibility of them, gets 404 rather than an empty list. Failing the task there would mean one such repository breaks an otherwise fine collection — the same failure shape as #9026.
### Performance
`loadIssueFieldValues` fetches the mapped fields for the connection in **one** query and indexes them by issue id, rather than one query per issue. Bounded by (issues × mapped fields), so at most 5 rows per issue.
Collection itself is one request per issue, which is inherent to the endpoint — it has no repo-wide variant. If you would prefer this go through `github_graphql` instead to batch it, say so and I will move it.
### Tests
14 unit tests, all passing:
- `issue_field_value_extractor_test.go` — `displayValue` across all five data types using the documented response bodies, including integral vs fractional numbers, `null`, and `multi_select` arriving without option objects; `scalarValue` on missing/unknown shapes.
- `issue_field_mapping_test.go` — mapping resolution and trimming, nil scope config, name lowercasing and dedup (one field driving two columns), case-insensitive and nil-safe lookup with no cross-issue leakage, date parsing across the documented and timestamp forms.
```
ok github.com/apache/incubator-devlake/plugins/github/tasks
ok github.com/apache/incubator-devlake/plugins/github/impl <- subtask graph is acyclic
ok github.com/apache/incubator-devlake/plugins/github/models
```
`migration-script-lint` passes (the script uses `archived.NoPKModel`, not `core/models/common`).
**I have not added an e2e dataflow test.** `NewDataFlowTester` panics without `E2E_DB_URL` and I have no DB here, so I would be shipping a test I could not run — every existing test in `plugins/github/e2e` fails the same way locally. Happy to add one with raw/snapshot CSV fixtures if you would like it; I would just be relying on CI to tell me it is right.
### Follow-ups
- Docs for the new scope config keys belong in `devlake-website`. Happy to send that PR alongside.
- Org-level field definitions (`GET /orgs/{org}/issue-fields`) are deliberately not collected: the value endpoint already returns `issue_field_name` and `data_type`, and skipping it avoids requiring org-level permissions. Easy to add if you want field descriptions or option colours in their own table.
- config-ui has no inputs for the new keys yet; they are settable through the API. Let me know if you want the UI in this PR or a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(github): collect GitHub issue fields and map them onto domain issues [devlake]
Posted by "vbhanuchander-lang (via GitHub)" <gi...@apache.org>.
vbhanuchander-lang commented on PR #9038:
URL: https://github.com/apache/devlake/pull/9038#issuecomment-5234867625
Pushed `18ec61c` adding the e2e dataflow test I said was missing. I ran a MySQL 8.4 container locally to set `E2E_DB_URL`, so this is verified rather than written blind.
`TestIssueFieldValueDataFlow` covers both halves.
**Extraction** — asserts the value normalisation per data type against the documented response shapes:
```
connection_id,issue_id,field_id,field_name,data_type,value,raw_value,option_color
1,346842831,123,Priority,single_select,Critical,"""Critical""",ff0000
1,346842831,456,Effort,number,5,5,
1,346842831,789,Target date,date,2024-12-31,"""2024-12-31""",
1,347255859,456,Effort,number,2.5,2.5,
1,347255859,901,Squad,multi_select,"backend,urgent","[""backend"",""urgent""]",
1,348630179,123,Priority,single_select,,null,
```
Note `5` rather than `5.0`, `2.5` keeping its precision, the multi_select joined while `raw_value` keeps the JSON array, and a null value producing an empty `value`.
**Conversion** — asserts the mapping reaches the domain issue, including the failure paths:
```
id,issue_key,priority,component,story_point,due_date
github:GithubIssue:1:346842831,5,Critical,,5,2024-12-31T00:00:00.000+00:00
github:GithubIssue:1:347255859,6,Low,"backend,urgent",2.5,
github:GithubIssue:1:348630179,7,,,,
```
Issue #7 is the interesting row: it carries `"soon"` in a date field and a null priority, and comes out with neither set — skipped with a warning rather than failing the task or writing a wrong value. The log confirms the path fires:
```
level=warning msg="issue #7: field \"Target date\" value \"soon\" is not a date, due date left unset"
```
The other 23 issues in the fixture are untouched, which is what verifies that an unmapped issue is left alone.
Whole plugin green with the database attached, including every pre-existing e2e test:
```
ok github.com/apache/incubator-devlake/plugins/github/e2e 3.667s
ok github.com/apache/incubator-devlake/plugins/github/impl
ok github.com/apache/incubator-devlake/plugins/github/tasks
ok github.com/apache/incubator-devlake/plugins/github/models
```
One fixture note in case it helps anyone else: the `params` column has to match the serialised form byte for byte. My first attempt used `json.dumps` defaults, which inserts spaces after separators, and the extractor silently found 0 rows because the `params = ?` clause did not match. Regenerating with compact separators fixed it, and the fixture is LF-terminated to match the existing ones.
Still happy to add config-ui inputs for the new scope config keys and the `devlake-website` docs, either here or as follow-ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(github): collect GitHub issue fields and map them onto domain issues [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh merged PR #9038:
URL: https://github.com/apache/devlake/pull/9038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(github): collect GitHub issue fields and map them onto domain issues [devlake]
Posted by "vbhanuchander-lang (via GitHub)" <gi...@apache.org>.
vbhanuchander-lang commented on PR #9038:
URL: https://github.com/apache/devlake/pull/9038#issuecomment-5247027197
`unit-test` is fixed in 7e57c1f.
`Test_GetPluginTablesInfo` compares the plugin's declared tables against the ones its migrations create, and I had added the new table without registering it:
```
table_info_test.go:121: The following tables are not returned by the TablesInfo method
_tool_github_issue_field_values
```
Added `GithubIssueFieldValue{}` to `Github.GetTablesInfo()`. I could not run that test on my machine — the `plugins` package needs libgit2 to build — so I ran it inside the `mericodev/lake-builder` image the job itself uses, rather than pushing and hoping:
```
ok github.com/apache/incubator-devlake/plugins 0.243s
```
Everything else in that run was already green, including the new e2e test on **both** engines:
| Check | Result |
|---|---|
| `e2e-mysql` | ✅ |
| `e2e-postgres` | ✅ |
| `lint` (golangci) | ✅ |
| `migration-script-lint` | ✅ |
| `check Apache license header` | ✅ |
| `commit-msg` | ✅ |
| `notice-year-check`, `grafana`, `yaml-lint` | ✅ |
Worth noting the e2e test passing on Postgres as well as MySQL, since the mapping query uses `LOWER(field_name) in ?` and I had only exercised it against MySQL locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org