Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]

["Ma77Ball (via GitHub)" <gi...@apache.org>]

GitHub user Ma77Ball closed a discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs

## Proposal: land "Modern Observability" (#4070) as a stack of small, self-contained PRs

Opening this to get feedback on the **approach and procedure** before raising the PRs.

### Problem
The observability work for #4070 grew into one large branch (~16k insertions across ~230 files): OTel bootstrap + log/metric/trace/profile pipelines, a tenant-scoped query gateway, an Angular dashboard, docker-compose backends — plus a codebase-wide logging/readability pass that rode along. That is far too big to review or merge safely as one PR.

### Approach
Decompose it into **26 small branches in two independent tracks**, each:
- **≤ ~1000 lines of logic**, compiles green on its own, and is **non-breaking** (the feature is default-off via `OTEL_SDK_DISABLED`, and the dashboard is health-gated), and
- adds one coherent slice toward the larger feature.

**Track A — the feature (18 stacked PRs):** foundations (log sanitizer → log bridge → OTel bootstrap → metrics → tracing) → deploy (compose/collector, Parca/eBPF) → query gateway (dtos → core → **logs / metrics / traces / profiles each its own PR**) → UI (shell → logs / metrics / traces / profiles panels).

**Track B — logging & a security fix (8 PRs, independent, off `main`):** human-readable lifecycle/error logging across engine, web layer, dashboard resources, workflow-core storage, the microservices, common/auth, common/dao; plus a fix to stop logging JWTs/headers/bodies in access-control.

### Procedure
- Each PR is tracked as a sub-issue of #4070 (#5327–#5352) and maps 1:1 to a branch.
- Track A PRs are **stacked** (each opens against the previous); Track B PRs target `main` directly and can merge in any order.
- Suggested land order: security fix + logging PRs (parallel) → the feature stack in numeric order. Shortest path to a first shippable signal (logs end-to-end): bootstrap → gateway-core → gateway-logs → ui-shell → ui-logs.
- Per-signal verification: backend via `sbt <module>/compile` (+ specs), frontend via `ng build`.

### Questions for reviewers
1. Is the two-track split + per-signal gateway/UI granularity the right reviewing unit, or would you prefer coarser PRs?
2. Should the codebase-wide logging track live under #4070, or as its own umbrella issue?
3. Any concerns with the stacked-PR procedure for the feature track?

GitHub link: https://github.com/apache/texera/discussions/5355

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org