You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "pballester (via GitHub)" <gi...@apache.org> on 2026/07/29 13:02:16 UTC
[I] [Bug][org] All org plugin API endpoints panic with nil pointer — root cause: routes bind handlers before InitPlugins runs [devlake]
pballester opened a new issue, #9021:
URL: https://github.com/apache/devlake/issues/9021
## What happened
Every `plugins/org` API endpoint (`teams.csv`, `users.csv`, `user_account_mapping.csv`, GET and PUT) returns HTTP 500 with a nil-pointer panic. Verified on v1.0.2 and v1.0.3-beta15 (helm deployment, MySQL backend). GET with `?fake_data=true` works (it never touches the store).
Panic (v1.0.3-beta15, `PUT /plugins/org/teams.csv`):
```
runtime error: invalid memory address or nil pointer dereference
/app/plugins/org/api/team.go:87 <- h.store.deleteAll(&crossdomain.Team{})
/app/server/api/router.go:149
```
GET without fake_data panics at `team.go:47` (`h.store.findAllTeams()`).
## Root cause (with evidence)
It's an initialization-order race between route registration and plugin initialization:
1. `runner.LoadGoPlugins` only does `Lookup("PluginEntry")` + `plugin.RegisterPlugin(...)` — it does **not** call `PluginInit.Init()` ([loader.go](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/core/runner/loader.go#L62-L83)).
2. `plugin.InitPlugins(basicRes)` — the only place `Init()` runs in server mode — is invoked from `pipelineServiceInit()` ([pipeline.go:83](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/server/services/pipeline.go#L83)).
3. `registerPluginEndpoints` builds the router from each plugin's `ApiResources()`. For `org`, that returns **bound method values** on instance state: `p.handlers.GetTeam`, etc. ([impl.go](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/plugins/org/impl/impl.go#L104-L120)).
4. In Go, a method value captures its receiver **at evaluation time**. If `ApiResources()` is evaluated before `Init()` set `p.handlers`, the router permanently holds handlers bound to a nil `*Handlers` — running `Init()` later doesn't rebind them.
Startup timeline from our pod logs proving the order (a DB migration was pending at boot, which delays `pipelineServiceInit` while the HTTP server must start to serve `/proceed-db-migration`):
```
12:13:58 plugin loaded org
12:14:11 endpoint PUT /plugins/org/teams.csv ... registerPluginEndpoints <- ApiResources() evaluated, p.handlers == nil
12:14:23 plugin: <X> doesn't implement 'PluginInit', it will be skipped. <- InitPlugins runs, 12s too late
```
`org` is the only plugin whose `ApiResources()` binds per-instance state created in `Init()` — other plugins read package-level vars at call time — which is why only `org` breaks, and why it breaks non-deterministically across deployments (whoever wins the race). This likely explains the long trail of closed-as-stale reports: #8590, #8271, #7957, #7219 (different symptoms of the same race — reporters couldn't pin it because it depends on startup timing; a pending DB migration at boot makes it deterministic).
Note the v1.0.3-beta fix `(&team{}).toDomainLayer(tt)` (team.go:84) resolved a *second, independent* nil deref, which made the remaining one harder to spot.
## What do you expect to happen
`plugins/org` CSV endpoints work regardless of startup timing.
## How to reproduce
1. Deploy v1.0.2 or v1.0.3-beta15 (helm chart, MySQL) with a pending DB migration at boot (e.g. any version upgrade), or any startup where route registration beats `pipelineServiceInit`.
2. Confirm the migration, then `curl -X PUT <endpoint>/plugins/org/teams.csv -F 'file=@teams.csv'` → HTTP 500, panic above. `GET /plugins/org/teams.csv` → same at team.go:47.
## Proposed fix (happy to submit a PR)
Two options, smallest first:
**A (org-scoped, minimal):** make `org`'s `ApiResources()` resolve handlers at call time instead of bind time — closures over `p` (pointer receiver) rather than method values over `p.handlers`:
```go
func (p *Org) ApiResources() map[string]map[string]plugin.ApiResourceHandler {
wrap := func(f func(*api.Handlers) plugin.ApiResourceHandler) plugin.ApiResourceHandler {
return func(in *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
if p.handlers == nil {
return nil, errors.Internal.New("org plugin not initialized yet")
}
return f(p.handlers)(in)
}
}
...
}
```
**B (framework):** guarantee `plugin.InitPlugins()` runs before `registerPluginEndpoints` (e.g. right after `runner.LoadPlugins` in `services.Init`), decoupling it from `pipelineServiceInit`. Fixes the class of bug for any future plugin with instance state, but touches startup ordering for everything.
## Version
v1.0.3-beta15 (also reproduced on v1.0.2). Helm chart on Kubernetes (EKS), external MySQL 8.0.
## Are you willing to submit a PR?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][org] All org plugin API endpoints panic with nil pointer — root cause: routes bind handlers before InitPlugins runs [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh closed issue #9021: [Bug][org] All org plugin API endpoints panic with nil pointer — root cause: routes bind handlers before InitPlugins runs
URL: https://github.com/apache/devlake/issues/9021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org