You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "Ma77Ball (via GitHub)" <gi...@apache.org> on 2026/06/03 05:26:37 UTC
[D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
GitHub user Ma77Ball created a discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
## Proposal: land "Modern Observability" (#4070) as a stack of small, self-contained PRs
Opening this to get feedback on the **approach and procedure** before raising the PRs.
### Problem
The observability work for #4070 grew into one large branch (~16k insertions across ~230 files): OTel bootstrap + log/metric/trace/profile pipelines, a tenant-scoped query gateway, an Angular dashboard, docker-compose backends — plus a codebase-wide logging/readability pass that rode along. That is far too big to review or merge safely as one PR.
### Approach
Decompose it into **26 small branches in two independent tracks**, each:
- **≤ ~1000 lines of logic**, compiles green on its own, and is **non-breaking** (the feature is default-off via `OTEL_SDK_DISABLED`, and the dashboard is health-gated), and
- adds one coherent slice toward the larger feature.
**Track A — the feature (18 stacked PRs):** foundations (log sanitizer → log bridge → OTel bootstrap → metrics → tracing) → deploy (compose/collector, Parca/eBPF) → query gateway (dtos → core → **logs / metrics / traces / profiles each its own PR**) → UI (shell → logs / metrics / traces / profiles panels).
**Track B — logging & a security fix (8 PRs, independent, off `main`):** human-readable lifecycle/error logging across engine, web layer, dashboard resources, workflow-core storage, the microservices, common/auth, common/dao; plus a fix to stop logging JWTs/headers/bodies in access-control.
### Procedure
- Each PR is tracked as a sub-issue of #4070 (#5327–#5352) and maps 1:1 to a branch.
- Track A PRs are **stacked** (each opens against the previous); Track B PRs target `main` directly and can merge in any order.
- Suggested land order: security fix + logging PRs (parallel) → the feature stack in numeric order. Shortest path to a first shippable signal (logs end-to-end): bootstrap → gateway-core → gateway-logs → ui-shell → ui-logs.
- Per-signal verification: backend via `sbt <module>/compile` (+ specs), frontend via `ng build`.
### Questions for reviewers
1. Is the two-track split + per-signal gateway/UI granularity the right reviewing unit, or would you prefer coarser PRs?
2. Should the codebase-wide logging track live under #4070, or as its own umbrella issue?
3. Any concerns with the stacked-PR procedure for the feature track?
GitHub link: https://github.com/apache/texera/discussions/5355
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
@zuozhiw Sounds good, I'll work on making each PR more self-contained going forward.
To answer your other question: the goal is to build our own UI and replace the LGTMP stack components with Apache-license-compatible alternatives. I've been working on this for a couple of weeks now, and it's starting to come together nicely. I also have branches in progress that expand logging coverage across the platform.
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17175097
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball closed a discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
## Proposal: land "Modern Observability" (#4070) as a stack of small, self-contained PRs
Opening this to get feedback on the **approach and procedure** before raising the PRs.
### Problem
The observability work for #4070 grew into one large branch (~16k insertions across ~230 files): OTel bootstrap + log/metric/trace/profile pipelines, a tenant-scoped query gateway, an Angular dashboard, docker-compose backends — plus a codebase-wide logging/readability pass that rode along. That is far too big to review or merge safely as one PR.
### Approach
Decompose it into **26 small branches in two independent tracks**, each:
- **≤ ~1000 lines of logic**, compiles green on its own, and is **non-breaking** (the feature is default-off via `OTEL_SDK_DISABLED`, and the dashboard is health-gated), and
- adds one coherent slice toward the larger feature.
**Track A — the feature (18 stacked PRs):** foundations (log sanitizer → log bridge → OTel bootstrap → metrics → tracing) → deploy (compose/collector, Parca/eBPF) → query gateway (dtos → core → **logs / metrics / traces / profiles each its own PR**) → UI (shell → logs / metrics / traces / profiles panels).
**Track B — logging & a security fix (8 PRs, independent, off `main`):** human-readable lifecycle/error logging across engine, web layer, dashboard resources, workflow-core storage, the microservices, common/auth, common/dao; plus a fix to stop logging JWTs/headers/bodies in access-control.
### Procedure
- Each PR is tracked as a sub-issue of #4070 (#5327–#5352) and maps 1:1 to a branch.
- Track A PRs are **stacked** (each opens against the previous); Track B PRs target `main` directly and can merge in any order.
- Suggested land order: security fix + logging PRs (parallel) → the feature stack in numeric order. Shortest path to a first shippable signal (logs end-to-end): bootstrap → gateway-core → gateway-logs → ui-shell → ui-logs.
- Per-signal verification: backend via `sbt <module>/compile` (+ specs), frontend via `ng build`.
### Questions for reviewers
1. Is the two-track split + per-signal gateway/UI granularity the right reviewing unit, or would you prefer coarser PRs?
2. Should the codebase-wide logging track live under #4070, or as its own umbrella issue?
3. Any concerns with the stacked-PR procedure for the feature track?
GitHub link: https://github.com/apache/texera/discussions/5355
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "zuozhiw (via GitHub)" <gi...@apache.org>.
GitHub user zuozhiw added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
OK, I'm a bit suspious on building our own FE, because searching and visualization do requires many work. But it's now AI era and we vibe code everything so it's fine.
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17198580
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
Hi @Ma77Ball, I don't have time to verify the project details yet, just a comment on opening issues and PRs related to a big topic. It will be good to have an umbrella issue (for example #4070) and on that issue to describe your plan. Then you can open multiple sub issues, each as a todo. The sub issues can be in parallel or in sequential to be completed.
For PRs though, if you find yourself stacking PRs (meaning that PR B depends on PR A, and PR C depends on PR B), then I suggest you open them one by one to make review easier:
- reviewer's attention needs to focus on incremental diff between the stacked PRs, and because github limitation, all PRs are against main branch, then PR B will contain diffs from PR A, and PR C will contain diffs from both A and B. It is hard for reviewers to check.
- for author, you, to resolve comments from PR A, you will likely need to push code and change the branch. Then you need to rebase or merge PR B and C, that's a lot of workload for author as well.
So my point is, you can open many issues (todos), but for PRs, it's better to open one at a time to get reviewed. If you have something that has no dependency and can be done/reviewed in parallel, sure you can open multiple PRs for those kind of parallel tasks.
I see you had more than 8(?) stacked PRs opening right now, and as a potential reviewer I am a bit lost where to start. for you, you need to update the other 7 as the first PR evolves, that's a lot of waste of time.
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17196834
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
@zuozhiw, any opinions on this path forward for raising the PRs for the observability task?
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17161147
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball edited a comment on the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
> [!NOTE]
> I plan to also move the tab under the admin panel
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17179197
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
I like the idea of building our own UI, more customizable.
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17200692
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
> [!NOTE]
> I plan to also move it under the admin panel
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17179197
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "zuozhiw (via GitHub)" <gi...@apache.org>.
GitHub user zuozhiw added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
Hi @Ma77Ball i like the approach of having two separate tracks, it looks good to me. I would say even if the diff is big, don't try to make too many Ps, eg 18 stacked PRs makes logic fragmented and might be even harder to review. I would encourage to make each pr to be more focused on a sub part but to be more complete. Current proposal sounds too fragmented to me.
Apart from that, I'm not following the context for the angular dashboard. Are we building our own UI for displaying observability stuff? I thought we are replying on third party vendors like grafana. Please let me know what do those changes include. Thx
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17174822
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball added a comment to the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
Here is what the design looks like now, but it can be edited based on input.
<img width="2560" height="1406" alt="image" src="https://github.com/user-attachments/assets/a80e0604-e0a5-489c-bc1b-d9724ec7dc06" />
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17179184
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
GitHub user Ma77Ball edited a comment on the discussion: Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs
@Yicong-Huang @chenlica @zuozhiw, any opinions on this path forward for raising the PRs for the observability task?
GitHub link: https://github.com/apache/texera/discussions/5355#discussioncomment-17161147
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org