You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/09/23 16:38:52 UTC
[GitHub] [lucene-solr] jpountz opened a new pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
jpountz opened a new pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917
This is called transitively from `DocumentsWriterFlushControl#doAfterDocument` which is synchronized and appears to be a point of contention.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494317183
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Oops thanks for catching. I thought they did because we make the assumption that all buffers have the same length in toDataInput, which is different from capacity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494311008
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Hmmm... Do they? I don't think this is the case, in general, since you can plug in an arbitrary block provider/ recycler?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494311008
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Hmmm... Do they? I don't think this is the case, in general, since you can plug in an arbitrary block provider/ recycler?
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
I'm not sure I like this assumption (that all blocks are equal). Maybe we could maintain a separate ram usage on block addition/ removal and thus make the sum-time constant but also accurate?
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Thanks Adrien!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494352956
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Thanks Adrien!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494317441
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Yep, I'll do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494314029
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
I'm not sure I like this assumption (that all blocks are equal). Maybe we could maintain a separate ram usage on block addition/ removal and thus make the sum-time constant but also accurate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917#discussion_r494317183
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Oops thanks for catching. I thought they did because we make the assumption that all buffers have the same length in toDataInput, which is different from capacity.
##########
File path: lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataOutput.java
##########
@@ -400,8 +400,13 @@ public void writeSetOfStrings(Set<String> set) {
public long ramBytesUsed() {
// Return a rough estimation for allocated blocks. Note that we do not make
// any special distinction for direct memory buffers.
- return RamUsageEstimator.NUM_BYTES_OBJECT_REF * blocks.size() +
- blocks.stream().mapToLong(buf -> buf.capacity()).sum();
+ ByteBuffer first = blocks.peek();
+ if (first == null) {
+ return 0L;
+ } else {
+ // All blocks have the same capacity.
Review comment:
Yep, I'll do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz merged pull request #1917: LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time.
Posted by GitBox <gi...@apache.org>.
jpountz merged pull request #1917:
URL: https://github.com/apache/lucene-solr/pull/1917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org