You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by GitBox <gi...@apache.org> on 2021/07/07 17:49:30 UTC
[GitHub] [orc] belugabehr opened a new pull request #739: Implement Improvements to ByteBufferAllocatorPool
belugabehr opened a new pull request #739:
URL: https://github.com/apache/orc/pull/739
TODO: CREATE JIRA
### What changes were proposed in this pull request?
Simplify ByteBufferAllocatorPool Key class. Cache hash value for Key as it's computed more than once: putBuffer to insert into Tree and getBuffer to remove it from tree.
### Why are the changes needed?
Less code. Performance.
### How was this patch tested?
No changes to functionality. Use existing unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #739:
URL: https://github.com/apache/orc/pull/739#issuecomment-900458753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] belugabehr commented on a change in pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r690380095
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
Excellent observation. Thanks. I will leave the current implementation in case the Collection ever changes down the road to a HashMap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #739: ORC-838: Simplify `compareTo/equals/putBuffer` of ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #739:
URL: https://github.com/apache/orc/pull/739#issuecomment-900461726
Thank you, @belugabehr , @kbendick , @guiyanakuang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on a change in pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
guiyanakuang commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r690537193
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
I also agree with leave equals and hashCode. Just putting the hashcode calculation in the constructor seems redundant in the current implementation.
ByteBufferAllocatorPool needs to support multiple calls to getBuffer with the same length before the buffer is returned for reuse. I think there may not be a suitable data structure other than TreeMap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] belugabehr commented on a change in pull request #739: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r665788810
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
- Key(int capacity, long insertionGeneration) {
+ Key (int capacity, long insertionGeneration) {
Review comment:
Nope. Not sure why I even did that. Thanks for pointing it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] kbendick commented on a change in pull request #739: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r665751393
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
- Key(int capacity, long insertionGeneration) {
+ Key (int capacity, long insertionGeneration) {
Review comment:
Nit: Is this extra space expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun merged pull request #739: ORC-838: Simplify `compareTo/equals/putBuffer` of ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun merged pull request #739:
URL: https://github.com/apache/orc/pull/739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on a change in pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
guiyanakuang commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r690537193
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
I also agree with leave equals and hashCode. Just putting the hashcode calculation in the constructor seems redundant in the current implementation.
ByteBufferAllocatorPool needs to support multiple calls to getBuffer with the same length before the buffer is returned for reuse. I think there may not be a suitable data structure other than TreeMap.
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
sorry, I didn't notice revert hash cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on a change in pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
guiyanakuang commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r690540573
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
sorry, I didn't notice revert hash cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #739:
URL: https://github.com/apache/orc/pull/739#issuecomment-900458753
Thank you for narrowing down the scoping by reverting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun merged pull request #739: ORC-838: Simplify `compareTo/equals/putBuffer` of ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun merged pull request #739:
URL: https://github.com/apache/orc/pull/739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on a change in pull request #739: ORC-838: Implement Improvements to ByteBufferAllocatorPool
Posted by GitBox <gi...@apache.org>.
guiyanakuang commented on a change in pull request #739:
URL: https://github.com/apache/orc/pull/739#discussion_r686775537
##########
File path: java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
##########
@@ -504,38 +504,33 @@ static void readDiskRanges(FSDataInputStream file,
private static final class Key implements Comparable<Key> {
private final int capacity;
private final long insertionGeneration;
+ private final int hash;
Key(int capacity, long insertionGeneration) {
this.capacity = capacity;
this.insertionGeneration = insertionGeneration;
+ this.hash = new HashCodeBuilder().append(capacity).append(insertionGeneration)
Review comment:
I found that the equals and hashCode methods of the Key are not used at all, because the TreeMap implementation logic only use of compareTo. It is redundant to put the hashCode calculation statement in the constructor, and in fact there is no need to override them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org