You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/11 02:27:05 UTC
[GitHub] [doris] liaoxin01 opened a new pull request, #13269: [enhancement](storage) set the segment cache capacity according to the open file limit of the process
liaoxin01 opened a new pull request, #13269:
URL: https://github.com/apache/doris/pull/13269
# Proposed changes
Issue Number: close #xxx
## Problem summary
Now segment_cache_capacity is 1000000, which usually exceeds open file limit. A large number of file descriptor will not be eliminated because of segment_cache_capacity is so large, resulting in too many open files errors. The client will receive -3109 error.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13269: [enhancement](storage) set the segment cache capacity according to the open file limit of the process
Posted by GitBox <gi...@apache.org>.
dataroaring merged PR #13269:
URL: https://github.com/apache/doris/pull/13269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13269: [enhancement](storage) set the segment cache capacity according to the open file limit of the process
Posted by GitBox <gi...@apache.org>.
dataroaring commented on code in PR #13269:
URL: https://github.com/apache/doris/pull/13269#discussion_r991762793
##########
be/src/runtime/exec_env_init.cpp:
##########
@@ -287,7 +287,20 @@ Status ExecEnv::_init_mem_tracker() {
<< PrettyPrinter::print(storage_cache_limit, TUnit::BYTES)
<< ", origin config value: " << config::storage_page_cache_limit;
- SegmentLoader::create_global_instance(config::segment_cache_capacity);
+ uint64_t fd_number = config::min_file_descriptor_number;
+ struct rlimit l;
+ int ret = getrlimit(RLIMIT_NOFILE, &l);
+ if (ret != 0) {
+ LOG(WARNING) << "call getrlimit() failed. errno=" << strerror(errno)
+ << ", use default configuration instead.";
+ } else {
+ fd_number = static_cast<uint64_t>(l.rlim_cur);
+ }
+ // SegmentLoader caches segments in rowset granularity. So the size of
+ // opened files will greater than segment_cache_capacity.
+ uint64_t segment_cache_capacity = fd_number / 3 * 2;
+ LOG(INFO) << "segment cache capacity: " << segment_cache_capacity;
Review Comment:
log out fd_number here and print segment_cache_capacity = fd_number / 3 * 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org