You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/01/31 01:03:15 UTC
[GitHub] [hbase] swaroopak opened a new pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
swaroopak opened a new pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110
…tryQueue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r373721679
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
Review comment:
Well, this is not thread-safe anyway, so even with your change it is still susceptible to overflows. Not your problem but just wanted to point out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] swaroopak commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
swaroopak commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r376082001
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
- if(heapSize < maxSize) {
+ long cbSize = cb.heapSize();
+ if(heapSize + cbSize < maxSize) {
queue.add(cb);
- heapSize += cb.heapSize();
+ heapSize += cbSize;
} else {
LruCachedBlock head = queue.peek();
if(cb.compareTo(head) > 0) {
heapSize += cb.heapSize();
- heapSize -= head.heapSize();
+ heapSize -= cbSize;
Review comment:
Oh yes, my bad!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] binlijin commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
binlijin commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r374547943
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
- if(heapSize < maxSize) {
+ long cbSize = cb.heapSize();
+ if(heapSize + cbSize < maxSize) {
Review comment:
format code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] swaroopak commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
swaroopak commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#issuecomment-583111098
@bharathv where should I add UT? I dont see an existing UT class for this class. Should I create a new one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] swaroopak commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
swaroopak commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r376081965
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
Review comment:
I was wondering if we should make the method synchronized and heapSize a volatile variable? What are your thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r376200409
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
Review comment:
That should be a separate change and is probably more involved ..just making it synchronized affects performance since that serializes all the invocations. (I'm not too familiar with that area of code on top of my head to explain the implications correctly, need to study it a little bit).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#issuecomment-580580134
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 55s | master passed |
| +1 :green_heart: | compile | 0m 58s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
| +0 :ok: | spotbugs | 4m 46s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 44s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 37s | the patch passed |
| +1 :green_heart: | compile | 0m 58s | the patch passed |
| +1 :green_heart: | javac | 0m 58s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 12s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 22s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
| +1 :green_heart: | findbugs | 4m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 155m 48s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 218m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1110 |
| JIRA Issue | HBASE-23761 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d58f3f64047e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1110/out/precommit/personality/provided.sh |
| git revision | master / 85b0c8e6bc |
| Default Java | 1.8.0_181 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/1/testReport/ |
| Max. process+thread count | 4472 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#issuecomment-580600688
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 0s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | checkstyle | 1m 14s | master passed |
| +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
| +0 :ok: | spotbugs | 4m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 3s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 15s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 22s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
| +1 :green_heart: | findbugs | 4m 52s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 33m 20s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 95m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.io.hfile.TestCachedBlockQueue |
| | hadoop.hbase.io.hfile.TestLruBlockCache |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1110 |
| JIRA Issue | HBASE-23761 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 06c5a7ac2e54 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1110/out/precommit/personality/provided.sh |
| git revision | master / 85b0c8e6bc |
| Default Java | 1.8.0_181 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/2/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/2/testReport/ |
| Max. process+thread count | 678 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1110/2/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1110: HBASE-23761: The new cache entry can overflow the maxSize in CachedEn…
URL: https://github.com/apache/hbase/pull/1110#discussion_r373720985
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruCachedBlockQueue.java
##########
@@ -65,14 +65,15 @@ public LruCachedBlockQueue(long maxSize, long blockSize) {
* @param cb block to try to add to the queue
*/
public void add(LruCachedBlock cb) {
- if(heapSize < maxSize) {
+ long cbSize = cb.heapSize();
+ if(heapSize + cbSize < maxSize) {
queue.add(cb);
- heapSize += cb.heapSize();
+ heapSize += cbSize;
} else {
LruCachedBlock head = queue.peek();
if(cb.compareTo(head) > 0) {
heapSize += cb.heapSize();
- heapSize -= head.heapSize();
+ heapSize -= cbSize;
Review comment:
why does this line change? IIUC, we are removing the current 'head' and adding the input cache block, so the existing code makes sense?
or did you mean to change the line above ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services