You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "lokidundun (via GitHub)" <gi...@apache.org> on 2026/03/21 07:27:14 UTC
[PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
lokidundun opened a new pull request, #719:
URL: https://github.com/apache/hugegraph-toolchain/pull/719
<!--
Thank you very much for contributing to Apache HugeGraph, we are happy that you want to help us improve it!
Here are some tips for you:
1. If this is your first time, please read the [contributing guidelines](https://github.com/apache/hugegraph/blob/master/CONTRIBUTING.md)
2. If a PR fix/close an issue, type the message "close xxx" (xxx is the link of related
issue) in the content, GitHub will auto link it (Required)
3. Name the PR title in "Google Commit Format", start with "feat | fix | perf | refactor | doc | chore",
such like: "feat(core): support the PageRank algorithm" or "fix: wrong break in the compute loop" (module is optional)
skip it if you are unsure about which is the best component.
4. One PR address one issue, better not to mix up multiple issues.
5. Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]` (or click it directly after
published)
-->
## Purpose of the PR
- close #xxx <!-- or use "fix #xxx", "xxx" is the ID-link of related issue, e.g: close #1024 -->
<!--
Please explain more context in this section, clarify why the changes are needed.
e.g:
- If you propose a new API, clarify the use case for a new API.
- If you fix a bug, you can clarify why it is a bug, and should be associated with an issue.
-->
## Main Changes
<!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. These change logs are helpful for better ant faster reviews.)
For example:
- If you introduce a new feature, please show detailed design here or add the link of design documentation.
- If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
- If there is a discussion in the mailing list, please add the link. -->
## Verifying these changes
<!-- Please pick the proper options below -->
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests here)*.
- [ ] Need tests and can be verified as follows:
- xxx
## Does this PR potentially affect the following parts?
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] Nope
- [ ] Dependencies (add/update license info) <!-- Don't forget to add/update the info in "LICENSE" & "NOTICE" files (both in root & dist module) -->
- [ ] Modify configurations
- [ ] The public API
- [ ] Other affects (typed here)
## Documentation Status
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `Doc - TODO` <!-- Your PR changes impact docs and you will update later -->
- [ ] `Doc - Done` <!-- Related docs have been already added or updated -->
- [ ] `Doc - No Need` <!-- Your PR changes don't impact/need docs -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #719:
URL: https://github.com/apache/hugegraph-toolchain/pull/719#discussion_r2971566565
##########
hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/service/load/FileMappingService.java:
##########
@@ -105,6 +105,12 @@ public List<FileMapping> listAll() {
return this.mapper.selectList(null);
}
+ public List<FileMapping> listByConnId(int connId) {
Review Comment:
not used now:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
Posted by "lokidundun (via GitHub)" <gi...@apache.org>.
lokidundun closed pull request #719: fix(hubble): fix loadfile limit
URL: https://github.com/apache/hugegraph-toolchain/pull/719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
Posted by "lokidundun (via GitHub)" <gi...@apache.org>.
lokidundun closed pull request #719: fix(hubble): fix loadfile limit
URL: https://github.com/apache/hugegraph-toolchain/pull/719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #719:
URL: https://github.com/apache/hugegraph-toolchain/pull/719#discussion_r2971569820
##########
hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/service/load/FileMappingService.java:
##########
@@ -105,6 +105,12 @@ public List<FileMapping> listAll() {
return this.mapper.selectList(null);
}
+ public List<FileMapping> listByConnId(int connId) {
+ QueryWrapper<FileMapping> query = Wrappers.query();
+ query.eq("conn_id", connId);
+ return this.mapper.selectList(query);
+ }
+
Review Comment:
`listByConnId()` is newly added but not referenced anywhere in the codebase (current usage sites still call `listAll()`/`list(...)`). If it’s not needed for this fix, consider removing it to avoid dead code; otherwise, please add the intended call site as part of this PR.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(hubble): fix loadfile limit [hugegraph-toolchain]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #719:
URL: https://github.com/apache/hugegraph-toolchain/pull/719#discussion_r2969318701
##########
hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/controller/load/FileUploadController.java:
##########
@@ -277,12 +277,12 @@ private void checkFileValid(int connId, int jobId, JobManager jobEntity,
long totalFileSizeLimit = this.config.get(
HubbleOptions.UPLOAD_TOTAL_FILE_SIZE_LIMIT);
- List<FileMapping> fileMappings = this.service.listAll();
+ List<FileMapping> fileMappings = this.service.listByConnId(connId);
long currentTotalSize = fileMappings.stream()
.map(FileMapping::getTotalSize)
.reduce(0L, (Long::sum));
Review Comment:
`currentTotalSize` is calculated by loading every `FileMapping` for the connection into memory and then reducing with boxed `Long`s. For large numbers of mappings this is unnecessarily expensive on every upload request. Consider using a DB-side `SUM(total_size)` aggregate (or at least `mapToLong(...).sum()`) to avoid allocating and iterating over full entities.
```suggestion
.mapToLong(FileMapping::getTotalSize)
.sum();
```
##########
hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/controller/load/FileUploadController.java:
##########
@@ -277,12 +277,12 @@ private void checkFileValid(int connId, int jobId, JobManager jobEntity,
long totalFileSizeLimit = this.config.get(
HubbleOptions.UPLOAD_TOTAL_FILE_SIZE_LIMIT);
- List<FileMapping> fileMappings = this.service.listAll();
+ List<FileMapping> fileMappings = this.service.listByConnId(connId);
long currentTotalSize = fileMappings.stream()
.map(FileMapping::getTotalSize)
.reduce(0L, (Long::sum));
Ex.check(fileSize + currentTotalSize <= totalFileSizeLimit,
- "load.upload.file.exceed-single-size",
+ "load.upload.file.exceed-total-size",
FileUtils.byteCountToDisplaySize(totalFileSizeLimit));
Review Comment:
The total upload limit is now computed by summing `total_size` across *all* `FileMapping` records for the connection. Since completed mappings are persisted (and `totalSize` is set on completion), this quota becomes cumulative across jobs and won’t “reset” when a new job is created, which seems to be the core problem described in #496. Consider scoping the sum to the current job (`conn_id` + `job_id`), or using `jobEntity.getJobSize()` (which is already maintained on upload/delete) so the limit is enforced per job and naturally resets between jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org