You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "kunwp1 (via GitHub)" <gi...@apache.org> on 2026/07/10 05:22:08 UTC
[PR] fix(frontend, v1.2): speed up dataset bulk upload with many files [texera]
kunwp1 opened a new pull request, #6313:
URL: https://github.com/apache/texera/pull/6313
### What changes were proposed in this PR?
Backport of #6306 to `release/v1.2` (cherry-pick of the four PR commits `3fd0068`, `b47c5a6`, `ade89a2`, `8e28e13`; one conflict in `frontend/src/jsdom-svg-polyfill.ts` — this branch carries an older revision of that file — resolved by re-adding the `ResizeObserver` stub in the branch's style).
This fixes bulk uploads of many files to a dataset being extremely slow (~1,100 files took ~12 minutes) because the Pending and Finished lists fully re-rendered on every change-detection pass on the same thread driving the uploads. The backport carries the complete fix from #6306: a Map-backed pending queue with a memoized `queuedFileNames` snapshot, `cdk-virtual-scroll` + `trackBy` on the Pending and staged-changes lists, an `auditTime`-coalesced staged-objects refetch (at most one dataset-diff request per second instead of one per finished file) with optimistic Finished-count updates so both headers move in real time, viewport re-measure on collapse-panel expand, and two upload-slot-leak fixes (`totalTime === 0` finishes and same-name re-uploads within the 5s hide window). See #6306 for the review discussion and a screen recording of a 1,100-file upload.
### Any related issues, documentation, discussions?
Backport of #6306, which fixes #5586 on `main` (as of this backport, #6306 is still awaiting merge).
### How was this PR tested?
The two component spec files shipped with the fix were run on this branch (`ng test --include '**/user-dataset-explorer/dataset-detail.component.spec.ts' --include '**/user-dataset-staged-objects-list.component.spec.ts'`): 26/26 passed. `yarn format:ci` passes. On #6306, Codecov reports all modified lines covered.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code, Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(frontend, v1.2): speed up dataset bulk upload with many files [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6313:
URL: https://github.com/apache/texera/pull/6313#issuecomment-4932256344
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@xuang7`, `@kz930`, `@Yicong-Huang`
You can notify them by mentioning `@xuang7`, `@kz930`, `@Yicong-Huang` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(frontend, v1.2): speed up dataset bulk upload with many files [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6313:
URL: https://github.com/apache/texera/pull/6313#issuecomment-4932285102
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/6313?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 52.32%. Comparing base ([`aaaa3d3`](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c88dff8`](https://app.codecov.io/gh/apache/texera/commit/c88dff83f7df004a1dc8b7fbe2c4d9a84942e713?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## release/v1.2 #6313 +/- ##
==================================================
+ Coverage 52.18% 52.32% +0.13%
Complexity 2486 2486
==================================================
Files 1066 1074 +8
Lines 41343 42121 +778
Branches 4427 4535 +108
==================================================
+ Hits 21576 22040 +464
- Misses 18497 18774 +277
- Partials 1270 1307 +37
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.61% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.46% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `1.65% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.06% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.59% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.13% <100.00%> (+0.57%)` | :arrow_up: | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.81% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.74% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/6313/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.69% <ø> (ø)` | | Carriedforward from [aaaa3d3](https://app.codecov.io/gh/apache/texera/commit/aaaa3d3995b232ad5f4e4481cbf58192e28c71a8?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/6313?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(frontend, v1.2): speed up dataset bulk upload with many files [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
xuang7 merged PR #6313:
URL: https://github.com/apache/texera/pull/6313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org