You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "mbaedke (via GitHub)" <gi...@apache.org> on 2023/07/14 10:35:32 UTC
[GitHub] [jackrabbit-oak] mbaedke opened a new pull request, #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
mbaedke opened a new pull request, #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] nfsantos commented on pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos commented on PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027#issuecomment-1640037684
Yes, I agree that there should be a maximum bound on the memory size for situations where this is run with very large heaps. I had not considered that case. Ok, I'll make a PR to reduce the minimum size of the memory buffers for cases with low heap and to enforce a reasonable upper limit.
I have done performance tests but it is not really clear what are the best settings, depends in many external factors. The auto-detect logic is just to make it work with reasonable performance in any setup without requiring manual configuration of this parameter. I expect that when performance is important, we will define this value explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] nfsantos commented on pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos commented on PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027#issuecomment-1640108749
I created this PR to adjust the upper and lower bounds of the auto-detected values.
https://github.com/apache/jackrabbit-oak/pull/1032
Could you please review? I cannot request reviewers (I'm not a committer to OAK).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke commented on pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "mbaedke (via GitHub)" <gi...@apache.org>.
mbaedke commented on PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027#issuecomment-1639764192
On Jenkins, this ran into an OutOfMemoryError: https://ci-builds.apache.org/blue/organizations/jenkins/Jackrabbit%2Foak-trunk-pr/detail/PR-1027/1/tests
It happened during the execution of this method: https://github.com/apache/jackrabbit-oak/blob/59f25b5ee73f711334c176ea16b60be81563a3fe/oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java#L230
,which does some suspicious looking memory calculations. @nfsantos, could you explain what is going on there?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke merged pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "mbaedke (via GitHub)" <gi...@apache.org>.
mbaedke merged PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] nfsantos commented on pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos commented on PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027#issuecomment-1639942852
@mbaedke
The PipelinedStrategy class uses a few pre-allocated internal buffers. The total size of these buffers is bounded by a configuration parameter, the "working memory". If this parameter is not set, it is set automatically based on the current heap size. This is what is happening in the tests, as by default, this parameter is not set. The logic for autodetecting the working memory is here:
https://github.com/apache/jackrabbit-oak/blob/59f25b5ee73f711334c176ea16b60be81563a3fe/oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java#L218-L227
In this particular case the heap size is very small, only 512MB, so the value derived from the heap size (Xmm - 2G) is negative. There is a check for this, that sets the working memory to a default minimum of 512MB, which is too much for a heap of 512MB. Here are the relevant logs from the tests:
```
10:54:28.401 INFO [main] PipelinedStrategy.java:221 Auto detecting working memory. Maximum heap size: 512 MB, selected working memory: -1536 MB
10:54:28.402 WARN [main] PipelinedStrategy.java:223 Working memory too low. Setting to minimum: 512 MB
```
I was not expecting that the unit tests would be run such a small heap. Increasing the heap to 1GB should be sufficient for this test to pass.
I think it should be possible to change the minimum default value to something smaller so that the unit tests can run with 512MB of heap. What is the minimum guaranteed heap size that we can assume for the unit tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mbaedke commented on pull request #1027: OAK-10350: Update spring-boot dependency to version 2.7.13
Posted by "mbaedke (via GitHub)" <gi...@apache.org>.
mbaedke commented on PR #1027:
URL: https://github.com/apache/jackrabbit-oak/pull/1027#issuecomment-1639981452
I think it should be possible to change the minimum default value to something smaller so that the unit tests can run with 512MB of heap. What is the minimum guaranteed heap size that we can assume for the unit tests?
I'm suprised as well, 512M is really low. I don't know much about the Jenkins setup. Maybe it would be a good idea to make #autodetectWorkingMemoryMB() a little smarter. Let's assume, for example, that the Jdk runs with 64G for good reason, because it's expected to run tasks that require a lot of memory on a regular basis. If PipelinedStrategy takes away 62G, that would not be helpful and prone to run into issues. In such cases, an upper bound would be reasonable. Did you run performance tests with various memory settings?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org