You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/16 02:41:17 UTC
[GitHub] [incubator-doris] xy720 opened a new pull request #3881: [BUG]Fix broker read buffer size from input stream
xy720 opened a new pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881
#3879
This commit fixs a bug that broker cannot read the full length of buffer size, when the buffer size is set larger than 128k.
This bug will cause the data size returned by pread request to be less than 128K all the time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #3881: [BUG]Fix broker read buffer size from input stream
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3881: [BUG]Fix broker read buffer size from input stream
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881#discussion_r441250595
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -550,19 +550,20 @@ public ByteBuffer pread(TBrokerFD fd, long offset, long length) {
currentStreamOffset, offset);
}
}
- byte[] buf;
+ ByteBuffer buf;
if (length > readBufferSize) {
- buf = new byte[readBufferSize];
+ buf = ByteBuffer.allocate(readBufferSize);
} else {
- buf = new byte[(int) length];
+ buf = ByteBuffer.allocate((int) length);
}
try {
- int readLength = fsDataInputStream.read(buf);
+ int readLength = readByteBufferFully(fsDataInputStream, buf);
if (readLength < 0) {
throw new BrokerException(TBrokerOperationStatusCode.END_OF_FILE,
"end of file reached");
}
- return ByteBuffer.wrap(buf, 0, readLength);
+ logger.debug("read buffer from input stream, buffer size:" + buf.capacity() + ", read length:" + readLength);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3881: [BUG]Fix broker read buffer size from input stream
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881#discussion_r440943167
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -550,19 +550,20 @@ public ByteBuffer pread(TBrokerFD fd, long offset, long length) {
currentStreamOffset, offset);
}
}
- byte[] buf;
+ ByteBuffer buf;
if (length > readBufferSize) {
- buf = new byte[readBufferSize];
+ buf = ByteBuffer.allocate(readBufferSize);
} else {
- buf = new byte[(int) length];
+ buf = ByteBuffer.allocate((int) length);
}
try {
- int readLength = fsDataInputStream.read(buf);
+ int readLength = readByteBufferFully(fsDataInputStream, buf);
if (readLength < 0) {
throw new BrokerException(TBrokerOperationStatusCode.END_OF_FILE,
"end of file reached");
}
- return ByteBuffer.wrap(buf, 0, readLength);
+ logger.debug("read buffer from input stream, buffer size:" + buf.capacity() + ", read length:" + readLength);
Review comment:
if (logger.isDebugEnable()) {
logger.debug("read buffer from input stream, buffer size: {}, read length: {}", buf.capacity(), readLength);
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3881: [BUG]Fix broker read buffer size from input stream
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881#discussion_r440736171
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -550,19 +550,20 @@ public ByteBuffer pread(TBrokerFD fd, long offset, long length) {
currentStreamOffset, offset);
}
}
- byte[] buf;
+ ByteBuffer buf;
if (length > readBufferSize) {
- buf = new byte[readBufferSize];
+ buf = ByteBuffer.allocate(readBufferSize);
} else {
- buf = new byte[(int) length];
+ buf = ByteBuffer.allocate((int) length);
}
try {
- int readLength = fsDataInputStream.read(buf);
+ int readLength = readByteBufferFully(fsDataInputStream, buf);
if (readLength < 0) {
throw new BrokerException(TBrokerOperationStatusCode.END_OF_FILE,
"end of file reached");
}
- return ByteBuffer.wrap(buf, 0, readLength);
+ logger.info("read length:" + length + ", readBufferSize:" + readBufferSize + ", return length:" + readLength);
Review comment:
Turn log level to debug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a change in pull request #3881: [BUG]Fix broker read buffer size from input stream
Posted by GitBox <gi...@apache.org>.
yiguolei commented on a change in pull request #3881:
URL: https://github.com/apache/incubator-doris/pull/3881#discussion_r440560907
##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -550,19 +550,20 @@ public ByteBuffer pread(TBrokerFD fd, long offset, long length) {
currentStreamOffset, offset);
}
}
- byte[] buf;
+ ByteBuffer buf;
if (length > readBufferSize) {
- buf = new byte[readBufferSize];
+ buf = ByteBuffer.allocate(readBufferSize);
} else {
- buf = new byte[(int) length];
+ buf = ByteBuffer.allocate((int) length);
}
try {
- int readLength = fsDataInputStream.read(buf);
+ int readLength = readByteBufferFully(fsDataInputStream, buf);
if (readLength < 0) {
throw new BrokerException(TBrokerOperationStatusCode.END_OF_FILE,
"end of file reached");
}
- return ByteBuffer.wrap(buf, 0, readLength);
+ logger.info("read length:" + length + ", readBufferSize:" + readBufferSize + ", return length:" + readLength);
Review comment:
Maybe too many logs here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org